Importing Standard Libraries¶
In [1]:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
Reading the Data¶
In [2]:
df = pd.read_csv('vehicle-1.csv')
data = df.copy()
df.head()
Out[2]:
Name of the Variables and Shape of the Data¶
In [3]:
df.columns
Out[3]:
In [4]:
df.shape
Out[4]:
Exploratory Data Analysis¶
In [5]:
df.info()
In [6]:
df.isnull().sum()
Out[6]:
Observation¶
- The Dataset consists of 19 variables with 846 rows
- Except for class column which is a Categorical column every other column is a Numerical column
- The Dataset has missing values for multiple columns. The columns are circularity, distance_circularity, radius_ratio, pr.axis_aspect_ratio, scatter_ratio, elongatedness, pr.axis_rectangularity, scaled_variance, scaled_variance.1, scaled_radius_of_gyration, scaled_radius_of_gyration.1, skewness_about, skewness_about.1 and skewness_about.2
- Class column is the Target variable & should not be considered for PCA
Handling Categorical Column¶
In this case it is the Target column i.e. Class Column
In [7]:
TARGET = "class"
df[TARGET].value_counts()
Out[7]:
In [8]:
from sklearn.preprocessing import LabelEncoder
labelencoder = LabelEncoder()
df[TARGET] = labelencoder.fit_transform(df[TARGET])
df[TARGET].value_counts()
Out[8]:
Observation: After label encoding we identify 0 = bus, 1 = car & 2 = van¶
Handling Missing Values¶
Since the number of missing values in not too large. We will replace the missing values with the MEDIAN of the column
In [9]:
from sklearn.impute import SimpleImputer
imputer = SimpleImputer(missing_values=np.nan, strategy='median', verbose=1)
#fill missing values with mean column values
transformed_values = imputer.fit_transform(df)
column = df.columns
df = pd.DataFrame(transformed_values, columns = column )
In [10]:
print("Data after treating missing value : ")
df.isnull().sum()
Out[10]:
In [11]:
df.describe().transpose()
Out[11]:
Observations¶
- Since for majority of the columns the mean and median (50%) are almost similar, it signifies that they are normally distributed and their are no outliers present in these variables.
- Few columns like scatter_ratio, scaled_variance & scaled_variance.1 indicates presence of outliers (to be treated later) hence there will be some skewness present in these variables. We will evaluate it later in detail.
In [12]:
sns.pairplot(df, diag_kind='kde')
Out[12]:
Inferences¶
- We can see that there are many features which are highly correlated (positively as well as negatively) among each other. The columns compactness, circularity, distance_circularity, radius_ratio, scatter_ratio, elongatedness, pr.axis_rectangularity, max.length_rectangularity, scaled_variance, scaled_variance.1, scaled_radius_of_gyration are highly correlated.
- Columns pr.axis_aspect_ratio and max.length_aspect_ratio do not have correlation with majority of other columns.
- Columns scaled_radius_of_gyration.1 has high negative correlation with skewness_about.2 and hollows_ratio columns. It has very less corelation with other columns.
- Column skewness_about.2 is positively correlated with the hollows_ratio column.
- We also notice presence of outliers in some dimensions as the points seems far from othe cluster of points.
- To summarize, there are lot of dimensions which seems correlated to each other visually and we will need to perform Correlation analysis to make more inferences about the data.
Numerical Variables¶
In [13]:
#Method to show Distribution & Box plot for the variable along with skewness
def showPlots(df, col):
fig,(ax1,ax2) = plt.subplots(nrows=1,ncols=2)
fig.set_size_inches(10,2)
sns.distplot(df[col],ax=ax1)
ax1.set_title("Distribution Plot")
sns.boxplot(df[col],ax=ax2)
ax2.set_title("Box Plot")
print(df[col].skew())
In [14]:
showPlots(df, 'compactness')
Observation: compactness column seems normally distributed.¶
In [15]:
showPlots(df, 'circularity')
Observation: circularity column seems normally distributed with multiple guassian's.¶
In [16]:
showPlots(df, 'distance_circularity')
Observation: distance_circularity column seems normally distributed with multiple guassian's.¶
In [17]:
showPlots(df, 'radius_ratio')
Observation: radius_ratio column is right skewed with outliers.¶
In [18]:
showPlots(df, 'pr.axis_aspect_ratio')
Observation: pr.axis_aspect_ratio column is right skewed with outliers.¶
In [19]:
showPlots(df, 'max.length_aspect_ratio')
Observation: max.length_aspect_ratio column is right skewed with outliers.¶
In [20]:
showPlots(df, 'scatter_ratio')
Observation: scatter_ratio column seems normally distributed with multiple guassian's.¶
In [21]:
showPlots(df, 'elongatedness')
Observations: elongatedness column seems normally distributed with multiple guassian's.¶
In [22]:
showPlots(df, 'pr.axis_rectangularity')
Observations: pr.axis_rectangularity column seems normally distributed with multiple guassian's.¶
In [23]:
showPlots(df, 'max.length_rectangularity')
Observations: max.length_rectangularity column seems normally distributed.¶
In [24]:
showPlots(df, 'scaled_variance')
Observations: scaled_variance column seems normally distributed with multiple guassian's.There is a outlier present.¶
In [25]:
showPlots(df, 'scaled_variance.1')
Observations: scaled_variance.1 column is lightly right skewed with outliers.¶
In [26]:
showPlots(df, 'scaled_radius_of_gyration')
Observations: scaled_radius_of_gyration column seems normally distributed.¶
In [27]:
showPlots(df, 'scaled_radius_of_gyration.1')
Observations: scaled_radius_of_gyration.1 column is highly right skewed with outliers.¶
In [28]:
showPlots(df, 'skewness_about')
Observations: skewness_about column is moderately right skewed with outliers.¶
In [29]:
showPlots(df, 'skewness_about.1')
Observations: skewness_about.1 column is lighty right skewed with an outlier.¶
In [30]:
showPlots(df, 'skewness_about.2')
Observations: skewness_about.2 column seems normally distributed.¶
In [31]:
showPlots(df, 'hollows_ratio')
Observations: hollows_ratio column seems normally distributed with multiple guassian's.¶
Let see the summary of the data to further understand the numerical variables.¶
Observations:¶
From the above graphs, it can be seen that the columns radius_ratio, pr.axis_aspect_ratio, max.length_aspect_ratio, scaled_variance, scaled_variance.1, scaled_radius_of_gyration.1, skewness_about, skewness_about.1 have outliers. Therefore, we will treat outliers of these columns before proceeding further.
Outliers Treatment¶
In [32]:
#We will handle the outliers using IQR for all the columns
from scipy.stats import iqr
def handleOutliers(odf):
Q1 = odf.quantile(0.25)
Q3 = odf.quantile(0.75)
IQR = Q3 - Q1
cleandf = odf[~((odf < (Q1 - 1.5 * IQR)) | (odf > (Q3 + 1.5 * IQR))).any(axis=1)]
print(cleandf.shape)
return cleandf
In [33]:
df = handleOutliers(df)
Target Column Distribution¶
In [34]:
df.hist(column=TARGET)
Out[34]:
Observation¶
- The columns "class" has categorical data having 3 different values of 0 for bus, 1 for car & 2 for van.
- The graph shows the distribution of the data across the three classes. Class type car i.e 1 takes up 50% of the data within the Vehicle Data Set. While van and bus have near about similar counts within the data set.
Correlation Comparison¶
In [35]:
corr = df.corr()
# select only the lower triangle of the correlation matrix
lower_triangle = np.tril(corr)
# to mask the upper triangle in the following heatmap
mask = lower_triangle == 0
plt.figure(figsize = (25,14))
sns.set(font_scale=1.8)
# Setting it to white so that we do not see the grid lines
sns.set_style(style = 'white')
sns.heatmap(lower_triangle, center=0.5, cmap= 'coolwarm', annot= True, xticklabels = corr.index, yticklabels = corr.columns,
cbar= False, mask = mask, linecolor='white', vmax=.8, fmt='.2f',linewidths=0.01)
#Logic to manage the known issue with Matplotlib version 3.1.11 w.r.t. Heat Map
b, t = plt.ylim() # discover the values for bottom and top
b += 0.5 # Add 0.5 to the bottom
t -= 0.5 # Subtract 0.5 from the top
plt.ylim(b, t) # update the ylim(bottom, top) values
plt.show()
sns.set(font_scale=1)
Observations¶
Strong or Very High Correlations ( > 0.9 )¶
Few important observations are
- circularity is correlated with max.length_rectangularity & scaled_radius_of_gyration
- distance_circularity is positively correlated with scatter_ratio & negtively with elongatedness
- scatter_ratio is correlated with scaled_variance.1, scaled_variance, pr.axis_rectangularity & negtively with elongatedness
- elongatedness is negatively correlated to scaled_variance.1, scaled_variance & pr.axis_rectangularity
- pr.axis_rectangularity is correlated with scaled_variance.1 & scaled_variance
- scaled_variance is correlated with scaled_variance.1
- scaled_radius_of_gyration.1 is negatively correlated with hollows_ratio & skewness_about.2
High Correlations ( > 0.7 till 0.9 )¶
Few important observations are
- The columns compactness, circularity, distance_circularity, radius_ratio, scatter_ratio, elongatedness, pr.axis_rectangularity, max.length_rectangularity, scaled_variance, scaled_variance.1 & scaled_radius_of_gyration are highly correlated among each other.
- skewness_about.2 is highly correlated with hollows_ratio
Low or No Correlations ( < 0.5 )¶
Few important observations are
- pr.axis_aspect_ratio, max_length_aspect_ratio & scaled_radius_of_gyration seems to have very little correlation with all the other columns
- scaled_radius_of_gyration & scaled_radius_of_gyration.1 seems to have almost no correlation
- scaled_radius_of_gyration.1 & skewness_about seems to have almost no correlation
- skewness_about, skewness_about.1 & skewness_about.2 are not correlated to each other.
Important Insight¶
- The Pair plot as well as the Correlation Comparison helps us to infer that there are considerable number of columns that are strongly correlated to each other. So, either they need to dropped of treated carefully before we go for model building.
- Our objective is to recognize whether the class is a van or bus or car based on some input features hence we need to ensure that there is little or no multicollinearity between the features. If our dataset has perfectly positive or negative attributes as can be obseverd from our correlation analysis, there is a high chance that the performance of the model will be impacted by Multicollinearity eventually leading to skewed or misleading results.
Resolve Multicollinearity¶
We are only considering a couple of most popular approaches used for resolving multicollinearity from the dataset
Approach 1¶
- If two features are highly correlated then there is no point using both features & we can drop one feature. Based on above observations we can decide to get rid of those columns whose correlation is +-0.9 or above. There are 8 such columns:
- max.length_rectangularity
- scaled_radius_of_gyration
- skewness_about.2
- scatter_ratio
- elongatedness
- pr.axis_rectangularity
- scaled_variance
- scaled_variance.1
Approach 2¶
- As we observed in our analysis that there are more than 50% attributes that are highly correlated with each other. Dropping or eliminating highly correlated columns could result in loss of information and hence we could prefer using the popular dimension reduction algorithm such as Principal Component Analysis (PCA)
We will be implementing both the approaches and finally evaluate them together.¶
Storing the processed dataset¶
In [36]:
processed_df = df
Model Building¶
In [37]:
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from prettytable import PrettyTable
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
Common Methods¶
Model Execution Functions¶
In [38]:
SPLIT_VALUE = 0.30
SEED = 1501
# Function to split Target Variable from data
def SplitData(d):
#Set of Independent Variables
X=d.drop(TARGET, axis=1)
#Dependent Variable
y=d[TARGET]
return X,y
# Function to Scale data
def ScaleData(X):
scaler = preprocessing.StandardScaler()
return scaler.fit_transform(X)
# Function to split data into training & test set
def SplitTrainTestData(X, y):
Xtrain, Xtest, ytrain, ytest = train_test_split (X, y, test_size=SPLIT_VALUE, stratify=y, random_state=SEED)
print("Training Data Shape: {0}".format(Xtrain.shape))
print("Testing Data Shape: {0}".format(Xtest.shape))
return Xtrain, Xtest, ytrain, ytest
# Function to fit the model
def ModelFit (model, Xtr, ytr):
fit = model.fit(Xtr, ytr)
print(fit)
return model
# Function to predict from model
def ModelPredict (model, Xtt, ytt):
pred = model.predict(Xtt)
acc_scr = accuracy_score(ytt, pred)
return pred, acc_scr
# Function to print the Results of the model
def PrintResults(model, pred, Xtr, Xtt, ytr, ytt):
x = PrettyTable()
x.field_names = ["Metrics", "Results"]
x.add_row(["Classification Report", classification_report(ytt, pred)])
x.add_row(["Accuracy Score", accuracy_score(ytt, pred)])
x.add_row(["",""])
x.add_row(["Confusion Matrix", confusion_matrix(ytt, pred)])
x.add_row(["",""])
x.add_row(["Training Data Score", model.score(Xtr, ytr)])
x.add_row(["",""])
x.add_row(["Testing Data Score", model.score(Xtt, ytt)])
print(x)
Hyper Parameter Tuning Function¶
In [39]:
def SVCTuneHyperParams(svc, Xtr, ytr):
svc = SVC()
Cs = [0.1, 1, 10, 100]
gammas = [0.01, 0.1, 1, 10]
kernel = ['linear', 'rbf', 'poly']
param = dict(kernel = kernel, C = Cs, gamma = gammas)
gs = GridSearchCV(svc, param, cv=3, scoring='accuracy', n_jobs = -1)
gs.fit(Xtr, ytr)
svc_bestScore = gs.best_score_
svc_bestParam = gs.best_params_
#Creating new model with best Parameters and running on the data again
k = svc_bestParam['kernel']
C = svc_bestParam['C']
g = svc_bestParam['gamma']
svc = SVC(kernel = k, C=C, gamma =g, probability=True)
svc.fit(Xtr, ytr)
x = PrettyTable()
x.field_names = ["Hyper Tuning", "Results"]
x.add_row(["Best Accuracy", svc_bestScore])
x.add_row(["",""])
x.add_row(["Best Parameter", svc_bestParam])
print(x)
return svc
Cross Validation Function¶
In [40]:
from sklearn import model_selection
SPLITS = 10
def KFoldCrossValidation (name, model, X, y, scoring):
kfold = model_selection.KFold(n_splits=SPLITS)
results = model_selection.cross_val_score(model, X, y, cv=kfold, scoring=scoring)
print(results)
mean, std = results.mean(), results.std()
x = PrettyTable()
x.field_names = ["Cross Validation", "Score Mean", "Score Standard Deviation"]
x.add_row([name, mean, std])
print(x)
return mean, std
Model Evaluation Function¶
In [41]:
# A single Function to execute all the steps for Model Evaluation
def ModelEvaluation(X, y, name, model, hyperTuneFunc, scoring):
# Creating Training & Test Set
Xtrain, Xtest, ytrain, ytest = SplitTrainTestData(X, y)
# Standardize X Data
Xtrain = ScaleData(Xtrain)
Xtest = ScaleData(Xtest)
# Model Training/Fitting
model = ModelFit(model, Xtrain, ytrain)
# Get Model Prediction & Accuracy Score
pred, scr = ModelPredict(model, Xtest, ytest)
# Results of model
PrintResults(model, pred, Xtrain, Xtest, ytrain, ytest)
# Perform Cross Validation
mean, std = KFoldCrossValidation(name, model, X, y, scoring)
# The code will tune the Hyper Parameter to retrieve the optimal value for the Model.
hy_model = hyperTuneFunc(model, Xtrain, ytrain)
# Get Model Prediction & Accuracy Score
hy_pred, hy_scr = ModelPredict(hy_model, Xtest, ytest)
# Results of model
PrintResults(hy_model, hy_pred, Xtrain, Xtest, ytrain, ytest)
return scr, mean, hy_scr
Import Libraries¶
In [42]:
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
Approach 1 - Drop Features¶
In [43]:
# Dropping the highly correlated columns from dataset
df = processed_df
droppedCols = ["max.length_rectangularity", "scaled_radius_of_gyration", "skewness_about.2",
"scatter_ratio", "elongatedness", "pr.axis_rectangularity", "scaled_variance", "scaled_variance.1"]
df1 = df.drop(droppedCols, axis=1)
# Splitting Data - Extract Target Column
X, y = SplitData(df1)
name = "Approach 1 - Drop Features"
model = SVC(kernel='linear', probability=True)
scoring = 'accuracy'
scr_a1, scr_cv_a1, scr_a1_hy = ModelEvaluation(X, y, name, model, SVCTuneHyperParams, scoring)
Approach 2 - PCA¶
Common Methods¶
In [44]:
from sklearn.decomposition import PCA
# Function to perform PCA analysis based on supplied parameters
def PCAFit(X, component):
# Standardize X Data
X = ScaleData(X)
# PCA technique implementation
pca = PCA(n_components = component)
pca.fit(X)
# Print Results
PrintPCAResults(pca)
return pca, X
# Function to print the results of PCA Fit
def PrintPCAResults(m):
x = PrettyTable()
x.field_names = ["PCA Analysis", "Result"]
x.add_row(["Eigen Values", m.explained_variance_])
x.add_row(["",""])
x.add_row(["Eigen Vectors", m.components_])
x.add_row(["",""])
x.add_row(["Variation Ratio", m.explained_variance_ratio_])
print(x)
def PCATransform(X, component):
#Perform PCA Fit
pca, X = PCAFit(X, component)
return pca.transform(X)
Approach 2 - Original Data¶
In [45]:
df = processed_df
# Splitting Original Data - Extract Target Column
X, y = SplitData(df)
name = "Approach 2 - Original Data"
model = SVC(kernel='linear', probability=True)
scoring = 'accuracy'
scr_a2, scr_cv_a2, scr_a2_hy = ModelEvaluation(X, y, name, model, SVCTuneHyperParams, scoring)
PCA¶
As a start point we will perform PCA Fit considering the components are same as the number of columns in the dataset
In [46]:
df = processed_df
# Splitting Original Data - Extract Target Column
X, y = SplitData(df)
pca, X = PCAFit(df, len(df.columns) - 1)
Principal Component Estimation¶
In [47]:
# Plotting the variance expalained by the principal components and the cumulative variance explained.
expVar = pca.explained_variance_ratio_
length = len(pca.explained_variance_ratio_) + 1
cExpVar = np.cumsum(pca.explained_variance_ratio_)
plt.figure(figsize=(10 , 5))
plt.bar(range(1, length), expVar, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, length), cExpVar, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Observation: The graph clearly shows that 7 Principal Components seems reasonable to capture more than 95% of the variance in data¶
Dimensionality Reduction¶
In [48]:
df = processed_df
# Splitting Original Data - Extract Target Column
X, y = SplitData(df)
pcadf = PCATransform(X, 7)
Pair Plot¶
Showing the pair plot of the PCA Tranform Data with only 7 Principal Components
In [49]:
pca = pd.DataFrame(pcadf)
sns.pairplot(pca, diag_kind='kde')
Out[49]:
Observation: The Pair plot clearly indicates that there is no correlation between the columns as expected after PCA¶
Approach 2 - PCA Data¶
In [50]:
name = "Approach 2 - PCA Data"
model = SVC(kernel='linear', probability=True)
scoring = 'accuracy'
scr_a2_pca, scr_cv_a2_pca, scr_a2_hy_pca = ModelEvaluation(pca, y, name, model, SVCTuneHyperParams, scoring)
Evaluate Performance of all Models¶
In [51]:
from prettytable import PrettyTable
x = PrettyTable()
x.field_names = ["Model Name", "Accuracy Score", "Cross Validation Score", "Accuracy Score - Hyper Tuning"]
x.add_row(["A1: Approach 1 - Drop Features", scr_a1, scr_cv_a1, scr_a1_hy])
x.add_row(["A2: Approach 2 - Original Data", scr_a2, scr_cv_a2, scr_a2_hy])
x.add_row(["A2-PCA: Approach 2 - PCA Data", scr_a2_pca, scr_cv_a2_pca, scr_a2_hy_pca])
print(x)
A1: Approach 1 - Drop Features :- In this case we have dropped the highly correlated columns from the dataset before Model Building¶
A2: Approach 2 - Original Data :- In this case we are using the original dataset for Model Building¶
A2-PCA: Approach 2 - PCA Data :- In this case we performed Dimensionality Reduction by PCA and used the data for Model Building¶
Conclusion¶
- According to the comparison table shown above A2 has performed the best. It has the highest accuracy score at almost 94%, cross validation score at 95% & accuracy score after hyper tuning the model at almost 96%.
- A1 has also performed fairly well considering we have dropped 8 correlated columns from the dataset. The accuracy score of almost 87% & cross validation score 88% are encouraging. The striking point is the accuracy score after hyper tuning which is very close to A2 ie. 95%. This indicates that if a well tuned model is selected then A1 can perform equally well despite loss of information.
- In A2-PCA model we were able to carry out dimensionality reduction successfully wherein we reduced the dataset from 18 to just 7 columns. The model has performed well with expected drop in accuracy but it also gives us many benefits like improved performance & efficiency, decrease requirement of memory, capacity & computation time and most important one is non collinearity of data which reduces overfitting & improves visualization.
- A2-PCA model has good accuracy score at almost 86% & Cross validation score at 85%. This model also jumped in accuracy after hyper tuning similar to A1. The accuracy improved to almost 94%. Here too appropriate model selection could make a difference.
- Overall A2-PCA model has delivered good results along with the noteworthy advantages that it bring to the table.
In [ ]: